Method and apparatus for synchronizing audio and video streams

ABSTRACT

Some embodiments of the invention provide a method for synchronizing an audio stream with a video stream. This method involves searching in the audio stream for audio data having values that match a distinct set of audio data values and synchronizing the audio stream with the video stream based on the search. In some embodiments, the distinct set of audio data values is defined by a predetermined distinct tone. In other embodiments, the distinct set of audio data values is defined by audio data contained in the video stream.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 10/407,954, filed Apr. 5, 2003, now issued as U.S.Pat. No. 7,142,250, which is incorporated herein by reference.

FIELD OF THE INVENTION

The invention is directed towards a method and apparatus forsynchronizing audio and video streams.

BACKGROUND OF THE INVENTION

Maintaining synchronization between audio and video streams is animportant aspect of a multimedia presentation. When content of audio andvideo components are captured, the signals are often processed inseparate pipelines, which typically causes the synchronizationrelationship between the audio and video components to be lost. Withouta proper synchronization relationship being re-established between thecaptured audio and video streams, there can be no guarantee that theaudio stream will correctly coincide with the video stream uponbroadcast or playback. Also, the divergence between the audio and videostreams may increase with the duration of a broadcast or playback.

The prior art does not provide a simple and effective technique forre-establishing a synchronous relationship between captured audio andvideo streams once the synchronous relationship between the streams hasbeen lost. Therefore, there is a need for a simple and effective methodthat re-establishes the synchronous relationship between captured audioand video streams.

SUMMARY OF THE INVENTION

Some embodiments of the invention provide a method for synchronizing anaudio stream with a video stream. This method involves searching in theaudio stream for audio data having values that match a distinct set ofaudio data values and synchronizing the audio stream with the videostream based on the search. In some embodiments, the distinct set ofaudio data values is defined by a predetermined distinct tone. In otherembodiments, the distinct set of audio data values is defined by audiodata contained in the video stream.

Some embodiments of the invention provide a method for determining acalculated sampling rate of audio samples contained in a video streamand applying the calculated sampling rate to an audio stream. The methodincludes receiving an audio stream of audio samples where the audiostream has an associated sampling rate. The method further includesreceiving a video stream containing video frames and audio samples, theaudio samples being duplicates of at least some of the audio samples inthe audio stream. The method finally includes determining a calculatedsampling rate of the audio samples in the video stream and modifying thesampling rate associated with the audio stream to match the calculatedsampling rate.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a conceptual diagram of an environment where someembodiments of the invention are implemented.

FIG. 2 illustrates conceptual diagrams of audio and video signals asthey are processed in the environment shown in FIG. 1.

FIG. 3 illustrates a diagram of data packets of multiplexed audio andvideo streams.

FIG. 4 illustrates a general process for synchronizing captured audioand video streams.

FIG. 5 illustrates a process of an editing application that causescapture of a video stream and an audio stream that includes a distincttone.

FIG. 6 illustrates a process of a video driver that causes a distincttone and video data to be sent to a capturing application.

FIG. 7 illustrates a process of an audio driver that causes a distincttone and audio data to be sent to a capturing application.

FIG. 8 illustrates a process of an editing application in analyzing acaptured audio stream to determine a synchronization offset.

FIG. 9 illustrates a conceptual diagram of an alternative environmentwhere some embodiments of the invention are implemented.

FIG. 10 illustrates conceptual diagrams of audio and video signals asthey are processed in the alternative environment shown in FIG. 9.

FIG. 11 illustrates a diagram of an audio data packet and a video datapacket of multiplexed audio and video streams.

FIG. 12 illustrates an alternative general process for synchronizingcaptured audio and video streams.

FIG. 13 illustrates a process of the AV editing application in analyzingcaptured audio and video streams to determine a synchronization offset.

FIG. 14 illustrates a process of the AV editing application in analyzinga captured video stream to determine a calculated sampling rate.

FIG. 15 presents a computer system with which some embodiments of theinvention are implemented.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, numerous details are set forth for purposeof explanation. However, one of ordinary skill in the art will realizethat the invention may be practiced without the use of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order not to obscure the description of theinvention with unnecessary detail.

Some embodiments of the invention provide a method for synchronizing anaudio stream with a video stream. This method involves (1) searching inthe audio stream for audio data having values that match a distinct setof audio data values and (2) synchronizing the audio stream with thevideo stream based on the search. In some embodiments, the distinct setof audio data values is defined by a predetermined distinct tone. Theseembodiments are described below in relation to section I. In otherembodiments, the distinct set of audio data values is defined by audiodata contained in the video stream. These embodiments are describedbelow in relation to section II.

I. FIG. 1 illustrates a conceptual diagram of an environment 100 wheresome embodiments of the invention are implemented. As shown in FIG. 1,the environment 100 includes an audio/video source (AV source) 105 and acomputer 110. The computer 110 contains a third party card 115, an audiodriver program 120, a video driver program 125, an audio/video capturingapplication (AV capturing application) 130, and an audio/video editingapplication (AV editing application) 135.

The AV source 105 provides audio and video signals (AV signals) to thecomputer 110. The AV source 105 can be any source providing AV signalssuch as a video tape deck, VCR, video camera, etc. The AV source 105 mayprovide high definition and uncompressed formats, digital or analog AVsignals, a single multiplexed stream of AV signals or a separate audiosignal and a separate video signal.

When the AV source 105 sends multiplexed AV signals to the computer 110(e.g., through a serial digital port), the third party card 115separates the multiplexed AV signals into discrete audio and videosignals. When the AV source 105 sends separate audio and video signalsto the computer 110 (e.g., through individual audio and video plugs),the third party card 115 passes the separate audio and video signals tothe audio driver program 120 and the video driver program 125,respectively.

The third party card 115 may be a video I/O board, an analog card, orthe like. The third party card 115 typically has audio and video inputsand outputs having, for example, serial digital video, composite video,or S video ports. Also, the third party card 115 can typically resampleaudio signals with non-standardized sampling rates that are receivedfrom the AV source 105. The third party card 115 can resample anon-standardized audio signal to a standardized sampling rate recognizedby the AV capturing application 130 and the AV editing application 135.Examples of companies making such third party cards are, for example,Pinnacle, Aurora, and Creative Labs.

Regardless of whether the third party card 115 receives multiplexed AVsignals or separate AV signals, the third party card 115 sends aseparate audio signal to the audio driver program 120 and a separatevideo signal to the video driver program 125. Typically, the AV signalshave a synchronous relationship when they are received by the thirdparty card 115. During the time between the separation of the audio andvideo signals by the third party card 115 and capture of the audio andvideo signals by the AV capturing application 130, however, thesynchronous relationship between the audio and video signals is lost.

After the third party card 115, the audio signal is processed by theaudio driver program 120 and the video signal is processed by the videodriver program 125. The audio and video driver programs 120 and 125 arehardware specific drivers that are configured to work specifically withthe third party card 115. The drivers provide an interface between thethird party card 115 and hardware and software resources of the computer110 (e.g., the AV capturing application 130 or the AV editingapplication 135) that may need to communicate with the third party card115.

The audio driver program 120 passes the audio signal to the AV capturingapplication 130 which captures the audio signal as an audio stream. Theaudio stream contains a series of audio data. Each instance of audiodata in the series of audio data is also known as an audio sample. Thevideo driver program 125 passes the video signal to the AV capturingapplication 130 which captures the video signal as a video stream. Thevideo stream is comprised of a series of video frames each containing aseries of video data. A synchronization relationship between the audioand video signals may have already been lost by the time the AVcapturing application 130 captures the audio and video signals.Therefore, without use of the present invention, the captured audio andvideo streams may not be in synchronization and playback or broadcast ofthe captured audio and video streams will produce a multimediapresentation that is not in synchronization.

The present invention provides a method for synchronizing the capturedaudio stream and video streams by having the third party card 115generate and send a distinct tone for capture in the audio stream by theAV capturing application 130. The length of the distinct tone provides asynchronization offset that the AV editing application 135 uses toadjust the captured audio and video streams to produce edited audio andvideo streams that are in synchronization. The AV capturing application130 may be any application capable of capturing or recording data ofaudio and video signals (e.g., QuickTime® by Apple Computer, Inc.). TheAV editing application 135 may be any application capable of editingcaptured audio and video streams (e.g., Final Cut Pro® by AppleComputer, Inc.).

FIG. 2 parts A through D illustrate conceptual diagrams of audio andvideo signals as they are processed in the environment shown in FIG. 1.Part A of FIG. 2 shows AV signals 205 that are sent by the AV source 105to the third party card 115. The AV signals 205 may be presented asmultiplexed AV signals or as separate audio and video signals. The videosignal portion of the AV signals 205 is comprised of a series of videoframes, each video frame being separated by a video frame boundary 206and containing a series of video data. The audio signal portion of theAV signals 205 is comprised of a series of audio frames, each audioframe being separated by an audio frame boundary 207 and containing aseries of audio data.

Between the audio and video signals 205 received by the third party card115, there typically is a synchronous relationship between each videoframe boundary 206 of the video signal and an audio frame of the audiosignal. After the third party card 115 receives the AV signals 205, thethird party card 115 sends a separate audio signal 210 to the audiodriver program 120 and a separate video signal 215 to the video driverprogram 125 (shown in part B of FIG. 2). During the separation of theaudio and video signals, any synchronization information between thesignals may be lost. Therefore, the AV editing application 135 will notassume that there is a synchronous relationship between a video frameboundary 206 of the video signal and an audio frame of the audio signal.

The AV capturing application 130 receives the separate audio signal 210from the audio driver program 120 and the separate video signal 215 fromthe video driver program 125. From the received signals, the AVcapturing application 130 produces a captured audio stream 220 and acaptured video stream 225 (shown in part C of FIG. 2). The capturedaudio stream 220 contains a series of audio data (each instance of audiodata being referred to as an audio sample). The captured video stream225 is comprised of a series of video frames each containing a series ofvideo data.

In some embodiments of the invention, the captured audio stream 220contains instances of distinct tone audio data 221 (i.e., distinct toneaudio samples) generated by the third party card 115. The length of thedistinct tone audio data 221 can be measured, for example, by the numberof audio data instances (i.e., audio samples) in the distinct tone audiodata 221. The length of the distinct tone audio data 221 can also bemeasured, for example, by a time duration corresponding to the number ofaudio data instances in the distinct tone audio data 221 (the timeduration also being determined by the sampling rate of the audio data).

The length of the distinct tone audio data 221 is referred to as thesynchronization offset 222. Using the synchronization offset 222, the AVediting application 135 synchronizes the captured audio stream 220 withthe captured video stream 225 to produce an edited audio stream 230 andan edited video stream 235 (shown in part D of FIG. 2) that are insynchronization with one another.

Typically, the AV capturing application 130 and the AV editingapplication 135 multiplex audio and video streams into one data stream.FIG. 3 illustrates a diagram of an audio data packet 310 and a videodata packet 315 of multiplexed audio and video streams 300 that form onedata stream. As shown in FIG. 3, the multiplexed audio and video streams300 is comprised of a header section 320, a resource data section 330,and interleaved audio and video data packets 310 and 315.

The header section 320 contains routing data such as origin anddestination information and the like. The resource data section 330typically contains data that imposes order or describes the audio orvideo data contained in the multiplexed audio and video streams 300. Forexample, the resource data section 330 may include data that indicatesthe sampling rate of the audio or video data contained in themultiplexed audio and video streams 300. In some embodiments of theinvention, the resource data section 330 utilizes synchronization offsetdata to synchronize the captured audio stream 220 with the capturedvideo stream 225.

Each of the audio and video data packets 310 and 315 in the multiplexedaudio and video streams 300 is comprised of a body section 325 thatcontains audio or video data. In the example shown in FIG. 3, the bodysection 325 of the audio data packet 310 contains audio data for 5 audioframes and the body section 325 of the video data packet 315 containsvideo data for 8 video frames.

FIG. 4 illustrates a general process 400 performed by components of thecomputer 110 for synchronizing captured audio and video streams.Initially, AV signals are received (at 405) by the third party card 115from the AV source 105. A user initiates (at 410) a capture procedure ofthe AV capturing application 130, for example, by selecting a captureoption of the AV capturing application 130 or the AV editing application135 through a graphical user interface (“GUI”). The user can interactwith the GUI through traditional GUI operations, such as clickoperations (e.g., to select an item), click-and-drag operations (e.g.,to move an item), etc.

The AV capturing application 130 receives and captures (at 415) the AVsignals from the third party card 115 to produce captured audio andvideo streams. The captured audio stream also includes a distinct tonegenerated by the third party card 115. The captured video stream iscomprised of a series of video frames each containing a series of videodata and the captured audio stream is comprised of a series of audioframes each containing a series of audio data (as described above inrelation to FIG. 3).

The user then ends (at 420) the capture procedure, for example, byselecting a capture ending option of the AV capturing application 130 orthe AV editing application 135 through the GUI. The AV editingapplication 135 then receives the captured audio and video streams fromthe AV capturing application 130 and analyzes (at 425) the capturedaudio stream to determine the length of the distinct tone in thecaptured audio stream. The length of the distinct tone is determined byperforming a search in the captured audio stream for audio data havingvalues that match a distinct set of audio data values determined by thedistinct tone. The length of the distinct tone corresponds to thesynchronization offset needed to synchronize the audio stream with thevideo stream. This process is described below in relation to FIG. 8.

Using the synchronization offset, the AV editing application 135synchronizes (at 430) the audio stream with the video stream to producean edited audio stream and an edited video stream that are insynchronization with one another. This can be achieved, for example, byaltering or adding data to the resource data section 330 of themultiplexed audio and video streams 300.

In one embodiment, data in the resource data section 330 can be alteredto indicate to an application receiving the edited audio and videostreams that playback of the audio stream is to start at a particularaudio data instance (i.e., audio sample) in the audio stream and thatall prior audio data instances is to be ignored. The particular audiodata instance is determined by the synchronization offset. For example,if the synchronization offset is 320, the data in the resource datasection 330 may indicate that playback of the audio stream will begin atthe 321^(st) audio data instance in the audio stream.

In an alternative embodiment, each audio data instance in the audiostream is re-ordered by subtracting the synchronization offset from anorder number of the audio data instance that indicates the position ofthe audio data instance in the audio stream. For example, if thesynchronization offset is 320, the 321^(st) audio data instance in theaudio stream may be re-ordered to be the 1^(st) audio data instance inthe audio stream, the 322^(nd) audio data instance in the audio streammay be re-ordered to be the 2^(nd) audio data instance in the audiostream, etc. Thus, an application receiving the edited audio and videostreams would begin playback of the audio stream at the 1^(st) audiodata instance (previously the 321^(st) audio data instance) in the audiostream.

After synchronizing (at 430) the audio stream with the video stream toproduce an edited audio stream and an edited video stream, the AVediting application 135 then records (at 435) the edited audio and videostreams that will be in synchronization when played back.

FIG. 5 illustrates a process of the AV editing application 135 thatcauses capture of a video stream and an audio stream that includes adistinct tone. This process begins when the user initiates (at 410) acapture procedure whereupon the AV editing application 135 receives (at505) a request to begin capture. The AV editing application 135 sends(at 510) a distinct tone command to the video driver program 125 which,in turn, sends the distinct tone command to the audio driver program120. In an alternative embodiment, the AV editing application 135 sendsthe distinct tone command directly to the audio driver program 120.

The distinct tone command is a command requesting a distinct tone havinga distinct set of audio data values. For example, the distinct set ofaudio data values may be a set of values that alternate in gain between+A dB and −A dB, where A is a real number. In one embodiment, thedistinct tone contains a distinct set of audio data values that wouldnormally not be present in the AV signals received from the AV source105 and is unique enough that it is highly unlikely another device wouldgenerate it. The distinct tone is generated by the third party card 115,for example, by on-board firmware or logic. In an alternativeembodiment, the audio driver program 120 may be configured to producethe distinct tone itself. If neither the third party card 115 nor theaudio driver program 120 can respond to the distinct tone command, thedistinct tone command will be ignored and capture of the audio and videosignals from the AV source 105 will proceed as normal (thus producingcaptured audio and video streams that are out of synchronization).

The distinct tone command also requires that the distinct tone be sentfor capture by the AV capturing application 130 until the video driverprogram 125 receives a first request for video data (as described belowin relation to FIG. 6). When the video driver program 125 receives such,the distinct tone command requires that the distinct tone no longer besent by the third party card 115 and that the audio and video data ofthe AV signals from the AV source 105 should be sent from the audiodriver program 120 and the video driver program 125, respectively, to becaptured as audio and video streams by the AV capturing application 130(as described below in relation to FIGS. 6 and 7). As such, the nextaudio data captured after the distinct tone audio data is captured willcorrespond to the video data of the first video frame of the capturedvideo stream. Therefore, the length of the distinct tone corresponds toa synchronization offset needed to offset the captured audio stream tosynchronize it with the captured video stream.

After sending (at 510) the distinct tone command, the AV editingapplication 135 then allocates (at 520) disk space in the computer 110to reserve data space for audio and video data to be captured. A requestfor audio data is then sent (525) to the audio driver program 120 inorder for audio data to pass through the audio driver program 120 to theAV capturing application 130. At this time, the third party card 115 isalready generating and sending the synch tone to the audio driverprogram 120 which is then passed onto the AV capturing application 130for capture in the audio stream. A request for video data is then sent(530) to the video driver program 125 in order for video data to passthrough the video driver program 125 to the AV capturing application130.

The AV editing application 135 receives (at 535) a request to endcapture when the user ends (at 420) the capture procedure, for example,by selecting a capture ending option of the AV capturing application 130or the AV editing application 135 through the GUI. A request is sent (at540) to the audio driver program 120 to stop sending audio data to theAV capturing application 130 for capture. A request is also sent (at545) to the video driver program 125 to stop sending video data to theAV capturing application 130 for capture.

FIG. 6 illustrates a process of the video driver program 125 that causesa distinct tone and video data to be sent to the AV capturingapplication 130. This process starts when the video driver program 125receives (at 605) a distinct tone command from the AV editingapplication 135. Upon receiving the distinct tone command, the videodriver program 125 sends the distinct tone command to the audio driverprogram 120. The distinct tone command notifies (at 610) the audiodriver program 120 to send a distinct tone from the third party card115. In an alternative embodiment, the AV editing application 135 sendsthe distinct tone command directly to the audio driver program 120. In afurther embodiment, the audio driver program 120 generates the distincttone rather than the third party card 115. The distinct tone commandalso requires that the distinct tone to be sent by the audio driverprogram 120 until the video driver program 125 receives a first requestfor video data. If the video driver program 125 has not received (at615—No) the first request for video data, the distinct tone willcontinue to be sent by the audio driver program 120.

When the video driver program 125 receives (at 615—Yes) the firstrequest for video data, the video driver program 125 notifies (at 620)the audio driver program 120 to stop sending the distinct tone and startsending audio data of the audio signal from the AV source 105. The videodriver program 125 also sends a request to the third party card 115 tostart sending video data from the video signal of the AV source 105 andsends (at 625) the video data to the AV capturing application 130. Thevideo driver program 125 continues to send video data until it hasreceived (at 630—Yes) a request from the AV editing application 135 tostop sending video data to the AV capturing application 130 for capture.

FIG. 7 illustrates a process of the audio driver program 120 that causesa distinct tone and audio data to be sent to the AV capturingapplication 130. This process starts when the audio driver program 120receives (at 705) a distinct tone command from the video driver program125, or in an alternative embodiment, from the AV editing application135. Upon receiving the distinct tone command, the audio driver program120 requests a distinct tone to be generated and sent from the thirdparty card 115. In a further embodiment, the audio driver program 120generates the distinct tone rather than the third party card 115. Ineither case, the audio driver program 120 sends (at 710) the distincttone to the AV capturing application 130 for capture.

If the audio driver program 120 has not received (at 715—No) a requestto stop sending the distinct tone, the audio driver program 120 willcontinue to send the distinct tone. When the audio driver program 120receives (at 715—Yes) a request to stop sending the distinct tone, theaudio driver program 120 stops (at 720) sending the distinct tone andstarts sending audio data of the audio signal from the AV source 105. Inorder to do so, the audio driver program 120 sends a request to thethird party card 115 to stop sending the distinct tone and start sendingthe audio data of the audio signal of the AV source 105. In analternative embodiment, the audio driver program 120 is generating thedistinct tone itself. Therefore, the audio driver program 120 would stopgenerating the distinct tone and would send a request to the third partycard 115 for audio data of the audio signal from the AV source 105. Theaudio driver program 120 continues to send audio data to the AVcapturing application 130 until it has received (at 725—Yes) a requestfrom the AV editing application 135 to stop sending audio data forcapture.

FIG. 8 illustrates a process of the AV editing application 135 inanalyzing a captured audio stream to determine a synchronization offset,the captured audio stream being comprised of a series of audio data. Theprocess starts when a template of audio data values that matches theaudio data values of the distinct tone is received (at 805). Thetemplate of audio data values is a distinct set of audio data valuesthat the process searches a match for in the audio stream. The templatehas a predetermined number of audio data values (one audio data valuefor each audio data instance) and may be received from the user orcreated by the AV editing application 135 itself using the audio datavalues of the distinct tone. If, for example, the distinct tone hasaudio data values that alternate between +A dB and −A dB, where A is areal number, the template may, for example, be comprised of four audiodata values: +A, −A, +A, and −A.

After receiving the template of audio data values, a template counter isset (at 810) to 0 and an audio data counter is set (at 815) to 0. Afirst audio data of a captured audio stream is then set (at 820) as acurrent audio data. In an alternative embodiment, any other audio dataof the captured audio stream is set (at 820) as the current audio data.The value of the current audio data is compared (at 825) to a firstaudio data value in the template of audio data values. In an alternativeembodiment, the current audio data value is compared (at 825) to anyother audio data value in the template of audio data values.

If the current audio data value does not match (at 830—No) the firstaudio data value in the template of audio data values, the next audiodata in the series of audio data of the captured audio stream is set (at835) as the current audio data and the audio data counter is increased(at 837). The audio data counter is then checked (at 840) to determineif it is greater than X, X being a predetermined integer value. Forexample, X may be set to equal the number of audio data instances (i.e.,audio samples) corresponding to 1 second of audio data in the capturedaudio stream, the actual value of X depending on the sampling rate ofthe audio stream. If it is determined (at 840—Yes) that the audio datacounter is greater than X, the process is aborted (at 845). Otherwise,the process continues and the value of the current audio data iscompared (at 825) to the first audio data value in the template of audiodata values.

If the current audio data value matches (at 830—Yes) the first audiodata value in the template of audio data values, the entire template ofaudio data values is then compared (at 850) to a correlating set ofaudio data values in the captured audio stream. For example, if thecurrent audio data value matches the first audio data value in thetemplate of audio data values and the template of audio data values iscomprised of 4 audio data values, the set of audio data values in thecaptured audio stream would be comprised of the current audio data valueand the next 3 audio data values in the captured audio stream. If thetemplate of audio data values does not match (at 850—No) the correlatingset of audio data values in the captured audio stream, the processcontinues at 835.

If the template of audio data values matches (at 850—Yes) thecorrelating set of audio data values in the captured audio stream, thetemplate counter is increased (at 855). The template of audio datavalues is then compared (at 860) to a next set of audio data values ofthe captured audio stream. In the example given above, the next set ofaudio data values would be comprised of the next 4 audio data values inthe captured audio stream. If the template of audio data values matches(at 860—Yes) the next set of audio data values of the captured audiostream, the template counter is increased (at 855).

If the template of audio data values does not match (at 860—No) the nextset of audio data values of the captured audio stream, thesynchronization offset is set (at 865) to equal the template countertimes the number of audio data values in the template of audio datavalues. In the example given above, if 3 sets of audio data values ofthe captured audio stream match the template of audio data values, thetemplate counter equals 3 and the synchronization offset equals 3*4 or12. Thus, the synchronization offset is approximately equal to thenumber of audio data instances (i.e., audio samples) in the capturedaudio stream that matches the audio data of the distinct tone.

II. FIG. 9 illustrates a conceptual diagram of an alternativeenvironment 900 where some embodiments of the invention are implemented.The environment 900 includes elements similar to the elements shown inthe environment 100 of FIG. 1. Only those aspects of the environment 900differing from the environment 100 of FIG. 1 will be described in detailhere.

As shown in FIG. 9, the computer 110 contains a demultiplexer 905(rather than a third party card 115 of the environment 100 of FIG. 1).The demultiplexer 905 receives multiplexed AV signals (e.g., through aserial digital port) from the AV source 105. The AV source 105 can beany source providing multiplexed AV signals such as a standard consumerDV, DV camera, DV video deck, etc.

After receiving the multiplexed AV signals, the demultiplexer 905 thenextracts and copies the audio signal portion of the multiplexed AVsignals to produce an audio only signal that is sent to the audio driverprogram 120. The demultiplexer 905 also sends a mixed video signal tothe video driver program 125, the mixed video signal being a duplicateof the multiplexed AV signals received from the AV source 105.

Typically, the multiplexed AV signals have a synchronous relationshipwhen received by the demultiplexer 905. When, however, the audio onlysignal is sent to the audio driver program 120 and the mixed videosignal is sent to the video driver program 125, there is no synchronousrelationship between the audio only signal and the mixed video signaleven though the audio only signal is a duplicate of an audio signalportion embedded in the mixed video signal.

After the demultiplexer 905, the audio only signal is processed by theaudio driver program 120 and the mixed video signal is processed by thevideo driver program 125. The audio driver program 120 passes the audioonly signal to the AV capturing application 130 which captures the audioonly signal as an audio stream. The captured audio stream contains aseries of audio data (each instance of audio data being referred to asan audio sample). The video driver program 125 passes the mixed videosignal to the AV capturing application 130 which captures the videosignal as a video stream. The captured video stream is comprised of aseries of audio samples and a series of video frames each containing aseries of video data. In the prior art, since there is no synchronousrelationship between the audio only signal and the mixed video signal,the captured audio and video streams will also be non-synchronous.

In some embodiments of the invention, the AV editing application 135extracts audio data from the video stream corresponding to apredetermined video frame in the video stream. The AV editingapplication 135 then performs a search in the audio stream for audiodata matching the extracted audio data and determines a total number ofprior audio data instances (i.e., audio samples) occurring before thematching audio data in the audio stream. The total number of prior audiodata instances (i.e., audio samples) is set as a synchronization offset.The AV editing application 135 offsets the audio stream by thesynchronization offset to synchronize the audio stream with the videostream and produce an edited audio stream and an edited video streamthat are in synchronization with one another.

In further embodiments of the invention, the AV editing application 135analyzes the captured video stream to determine a calculated samplingrate of the audio data embedded in the video stream. The AV editingapplication 135 then modifies the sampling rate associated with thecaptured audio stream to match the calculated sampling rate. Theseprocesses ensure that the sampling rate associated with the audio datain the audio stream matches the sampling rate of the audio data embeddedin the video stream.

FIG. 10 parts A through D illustrate conceptual diagrams of audio andvideo signals as they are processed in the alternative environment 900shown in FIG. 9. Part A of FIG. 10 shows multiplexed audio and videosignals (AV signals) 1005 that are sent by the AV source 105 to thedemultiplexer 905. The video signal portion of the multiplexed AVsignals 1005 is comprised of a series of video frames, each video framebeing separated by a video frame boundary 1006 and containing a seriesof video data. The audio signal portion of the multiplexed AV signals1005 is comprised of a series of audio frames, each audio frame beingseparated by an audio frame boundary 1007 and containing a series ofaudio data. Each video frame in the video signal portion has acoinciding audio frame in the audio signal portion that contains theaudio data for the video frame. So typically, between the multiplexed AVsignals 1005 received by the demultiplexer 905, there is a synchronousrelationship between each video frame boundary 1006 of the video signaland an audio frame of the audio signal.

The demultiplexer 905 receives the multiplexed AV signals 1005 andextracts and sends an audio only signal 1010 to the audio driver program120 and sends a mixed video signal 1015 to the video driver program 125(as shown in Part B of FIG. 10). The mixed video signal 1015 is aduplicate of the multiplexed AV signals 1005 show in Part A. Asdescribed above, there is no synchronous relationship between the audioonly signal 1010 and the mixed video signal 1015, so that a video frameboundary 1006 of the mixed video signal 1015 has no synchronousrelationship to an audio frame of the audio only signal 1010.

The AV capturing application 130 receives the audio only signal 1010from the audio driver program 120. From the received audio only signal1010, the AV capturing application 130 produces a captured audio stream1020 (shown in Part C of FIG. 10). The AV capturing application 130 alsoreceives the mixed video signal 1015 from the video driver program 125.From the received mixed video signal 1015, the AV capturing application130 produces a captured video stream 1025 (shown in Part C of FIG. 10)that contains audio data that are duplicates of at least some of theaudio data comprising the audio stream 1020.

In accordance with processes of the present invention, AV editingapplication 135 analyzes the captured audio stream 1020 and the capturedvideo stream 1025 to determine a synchronization offset. The AV editingapplication 135 uses the synchronization offset to synchronize thecaptured audio stream 1020 with the captured video stream 1025 toproduce an edited audio stream 1030 and an edited video stream 1035(shown in part D of FIG. 10) that are in synchronization with oneanother.

Typically, the AV capturing application 130 and the AV editingapplication 135 multiplex audio and video streams into one data stream.FIG. 11 illustrates a diagram of an audio data packet 310 and a videodata packet 315 of multiplexed audio and video streams 300 that form onedata stream. The elements of FIG. 11 are similar to the elements of FIG.3 except that a body section 1125 of each video data packet 315 containsaudio and video data at a predetermined ratio. This is due to the factthat the video stream 1025 and the edited video stream 1035 haveembedded audio data that has been captured from the mixed video signal1015. The body section 1125 contains audio and video data at apredetermined ratio of audio data to video data. In the example shown inFIG. 11, this ratio is 1 to 3.

FIG. 12 illustrates an alternative general process 1200 performed bycomponents of the computer 110 for synchronizing captured audio andvideo streams. Initially, multiplexed AV signals are received (at 1205)by the demultiplexer 905 from the AV source 105. A user initiates (at1210) a capture procedure of the AV capturing application 130. The AVcapturing application 130 receives and captures (at 1215) an audio onlysignal and a mixed video signal from the demultiplexer 905 to producecaptured audio and video streams, respectively. The captured videostream is comprised of a series of video frames each containing a seriesof video data and a series of audio samples. The captured audio streamis comprised of a series of audio frames each containing a series ofaudio samples, the captured audio stream having an associated samplingrate. The user then ends (at 1220) the capture procedure of the AVediting application 135.

The AV editing application 135 then receives the captured audio andvideo streams from the AV capturing application 130 and analyzes (at1225) the captured audio and video streams to determine asynchronization offset. The synchronization offset is determined byperforming a search in the captured audio stream for audio data havingvalues that match a distinct set of audio data values determined byaudio data embedded in the video stream. This process is described belowin relation to FIG. 13.

Using the synchronization offset, the AV editing application 135synchronizes (at 1230) the audio stream with the video stream. This canbe achieved, for example, by altering or adding data to the resourcedata section 330 of the multiplexed audio and video streams 300, asdiscussed above in relation to step 430 of FIG. 4.

The AV editing application 135 then analyzes (at 1235) the capturedvideo stream to determine a calculated sampling rate of the audio dataembedded in the video stream. This process is described below inrelation to FIG. 14. The AV editing application 135 modifies (at 1240)the sampling rate associated with the captured audio stream to match thecalculated sampling rate. This can be achieved, for example, by alteringdata in the resource data section 330 in the multiplexed audio and videostreams 300 to indicate the calculated sampling rate as the samplingrate of the captured audio stream 1020. In an alternative embodiment,the AV editing application 135 re-samples the audio stream at thecalculated sampling rate. The AV editing application 135 then records(at 1245) the edited audio and video streams that will be insynchronization when played back.

FIG. 13 illustrates a process of the AV editing application 135 inanalyzing captured audio and video streams to determine asynchronization offset. The captured audio stream is comprised of aseries of audio samples and the captured video stream is comprised of aseries of video frames each containing a series of video data and aseries of audio samples. The captured video stream contains audiosamples that are duplicates of at least some of the audio samplescomprising the captured audio stream.

Initially, a video frame counter is set (at 1305) to 1. A first videoframe of the captured video stream is then set (at 1310) as a currentvideo frame. In an alternative embodiment, any other first video frameof the captured video stream is set (at 1310) as the current videoframe. Audio data is then extracted (at 1315) from the current videoframe. The value of each instance of audio data (i.e., audio sample) inthe current video frame is used to create a current audio frame of audiodata values. The current audio frame of audio data values is a distinctset of audio data values that the process searches a match for in theaudio stream.

An audio data counter is then set (at 1320) to 0. A first audio data ofthe captured audio stream is set (at 1325) as a current audio data. Inan alternative embodiment, any other audio data of the captured audiostream is set (at 1325) as the current audio data. The value of thecurrent audio data is then compared (at 1330) to a first audio datavalue in the audio frame of audio data values. In an alternativeembodiment, the current audio data value is compared (at 1330) to anyother audio data value in the audio frame of audio data values.

If the current audio data value does not match (at 1330—No) the firstaudio data value in the audio frame of audio data values, the next audiodata in the series of audio data of the captured audio stream is set (at1335) as the current audio data and the audio data counter is increased(at 1337). The audio data counter is then checked (at 1340) to determineif it is greater than X, X being a predetermined integer value. Forexample, X may be set to equal the number of audio data instances (i.e.,audio samples) corresponding to 1 second of audio data in the capturedaudio stream, the actual value of X depending on the sampling rate ofthe captured audio stream. If it is determined (at 1340—No) that theaudio data counter is not greater than X, the value of the current audiodata is compared (at 1330) to the first audio data value in the audioframe of audio data values.

If it is determined (at 1340—Yes) that the audio data counter is greaterthan X, the next video frame in the series of video frames of thecaptured video stream is set (at 1345) as the current video frame andthe vide frame counter is increased (at 1347). The video frame counteris then checked (at 1350) to determine if it is greater than Y, Y beinga predetermined integer value. For example, Y may be set to equal thenumber of video frames corresponding to 1 second of video frames in thecaptured video stream, the actual value of Y depending on the frame rateof the captured video stream. If it is determined (at 1350—Yes) that thevideo frame counter is greater than Y, the process is aborted (at 1355).Otherwise, the process continues and audio data is extracted (at 1315)from the current video frame to create a current audio frame of audiodata values.

If the current audio data value matches (at 1330—Yes) the first audiodata value in the audio frame of audio data values, the entire audioframe of audio data values is then compared (at 1360) to a correlatingset of audio data values in the captured audio stream. For example, ifthe current audio data value matches the first audio data value in theaudio frame of audio data values and the audio frame of audio datavalues is comprised of 25 audio data values, the set of audio datavalues in the captured audio stream would be comprised of the currentaudio data value and the next 24 audio data values in the captured audiostream. If the audio frame of audio data values does not match (at850—No) the correlating set of audio data values in the captured audiostream, the process continues at 1335.

If the audio frame of audio data values matches (at 1360—Yes) thecorrelating set of audio data values in the captured audio stream, thesynchronization offset is set (at 1365) to equal the audio data counter.Thus, the synchronization offset is equal to a total number of prioraudio data instances (i.e., audio samples) in the audio stream thatoccur prior to the set of audio data in the audio stream having valuesthat match the audio frame of audio data values. The matching set ofaudio data in the audio stream is the corresponding audio data for aparticular video frame in the video stream that is identified by thevideo frame counter.

Therefore, the matching set of audio data in the audio stream should besynchronized with the particular video frame identified by the videoframe counter. This can be achieved, for example, by altering or addingdata to the resource data section 330 of the multiplexed audio and videostreams 300, as discussed above in relation to step 430 of FIG. 4. Ifthe video frame counter identifies the first video frame in the videostream, then the modifications to data in the resource data section 330need only affect the audio stream.

If, however, the video frame counter identifies a video frame other thanthe 1^(st) video frame in the video stream, the video stream must beoffset as well. This can be achieved using the same techniques discussedabove in relation to step 430 of FIG. 4. For example, data in theresource data section 330 can be altered to indicate to an applicationreceiving the edited audio and video streams that playback of the videostream is to start at a particular video frame in the video stream andthat all prior video frames are to be ignored. The particular videoframe is determined by the video frame counter. For example, if thevideo frame counter is 20, the data in the resource data section 330 mayindicate that playback of the video stream will begin at the 21^(st)video frame in the video stream.

FIG. 14 illustrates a process of the AV editing application 135 inanalyzing a captured video stream to determine a calculated samplingrate. The captured video stream is comprised of a series of video frameseach containing a series of video data and a series of audio data.

Initially, an audio data counter is set (at 1405) to 1 and a video framecounter is set (at 1410) to 1. A first video frame of the captured videostream is then set (at 1415) as a current video frame. In an alternativeembodiment, any other first video frame of the captured video stream isset (at 1415) as the current video frame. Audio data is then extracted(at 1420) from the current video frame. A first audio data (i.e., afirst audio sample) of the extracted audio data is then set (at 1425) asa current audio data. It is then checked (at 1430) to determine if thereis more extracted audio data. If so, the audio data counter is increased(at 1435) and the next audio data (i.e., next audio sample) of theextracted audio data is set (at 1440) as the current audio data.

If it is determined (at 1430—No) that there is no more extracted audiodata, it is checked (at 1445) if there are video frames remaining in thecaptured video stream. If so, the video frame counter is increased (at1450) and the next video frame of the captured video stream is set (at1452) as the current video frame.

If it is determined (at 1445—No) that there are no more video framesremaining in the captured video stream, a calculated sampling rate ofthe audio data in the video stream is determined (at 1455) using theaudio data counter (the total number of audio samples in the videostream) and the video frame counter (the total number of video frames inthe video stream). For example, the calculated sampling rate can bedetermined by the following equation:(total number of audio samples/total number of video frames)*apredetermined frame rate of the video stream.

To illustrate, if the total number of audio samples is equal to A, thetotal number of video frames is equal to B, and the predetermined framerate of the video stream is equal to 30 frames per second, then thecalculated sampling rate is equal to (A/B*30) audio samples per second.

FIG. 15 presents a computer system with which some embodiments of theinvention are implemented. Computer system 1500 includes a bus 1505, aprocessor 1510, a system memory 1515, a read-only memory 1520, apermanent storage device 1525, input devices 1530, and output devices1535.

The bus 1505 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 1500. For instance, the bus 1505 communicativelyconnects the processor 1510 with the read-only memory 1520, the systemmemory 1515, and the permanent storage device 1525.

The read-only-memory (ROM) 1520 stores static data and instructions thatare needed by the processor 1510 and other modules of the computersystem. The permanent storage device 1525, on the other hand, isread-and-write memory device. This device is a non-volatile memory unitthat stores instruction and data even when the computer system 1500 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 1525. Other embodiments use a removable storagedevice (such as a floppy disk or Zip® disk, and its corresponding diskdrive) as the permanent storage device. The permanent storage device maycontain, for example, instructions of applications such as the AVcapturing application 130 or the AV editing application 135 and data forcaptured audio and video streams.

Like the permanent storage device 1525, the system memory 1515 is aread-and-write memory device. However, unlike storage device 1525, thesystem memory is a volatile read-and-write memory, such as a randomaccess memory (RAM). The system memory stores some of the instructionsand data that the processor needs at runtime. In some embodiments, theinvention's processes are stored in the system memory 1515, thepermanent storage device 1525, and/or the read-only memory 1520.

Various embodiments of the invention may be implemented using thepermanent storage device 1525 or the system memory 1515. For example,analysis of audio and video data of captured audio and video streams (asdescribed above in relation to FIGS. 8, 13, and 14) may be performedafter such audio and video data have been written to the permanentstorage device 1525. Alternatively, such audio and video data may bemore efficiently analyzed while still in the system memory 1515 duringcapture by the AV capturing application 130.

From these various memory units, the processor 1510 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. For example, the processor 1510 may retrieveand execute instructions of the AV capturing application 130 or the AVediting application 135.

The bus 1505 also connects to the input and output devices 1530 and1535. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 1530 includealphanumeric keyboards and cursor-controllers. The output devices 1535display images generated by the computer system. For instance, thesedevices display IC design layouts. The output devices include printersand display devices, such as cathode ray tubes (CRT) or liquid crystaldisplays (LCD).

Finally, as shown in FIG. 15, bus 1505 also couples computer 1500 to anetwork 1565 through a network adapter (not shown). In this manner, thecomputer can be a part of a network of computers (such as a local areanetwork (“LAN”), a wide area network (“WAN”), or an Intranet) or anetwork of networks (such as the Internet). Any or all of the componentsof computer system 1500 may be used in conjunction with the invention.However, one of ordinary skill in the art would appreciate that anyother system configuration may also be used in conjunction with thepresent invention.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. Thus, one of ordinary skill in the artwould understand that the invention is not to be limited by theforegoing illustrative details, but rather is to be defined by theappended claims.

1. A method of processing media data, the method comprising: receivingan audio stream comprising a series of audio data; receiving a videostream comprising a series of video frames and a set of audio data; andsynchronizing the audio stream with the video stream based on the set ofaudio data in the video stream by (i) determining a calculated samplingrate based on the set of audio data in the video stream and (ii)resampling the audio stream at the calculated sampling rate.
 2. Themethod of claim 1 further comprising multiplexing the audio stream andthe video stream into a multiplexed media stream.
 3. The method of claim1, wherein synchronizing the audio stream with the video streamcomprises utilizing at least one counter for counting the set of audiodata in the video stream.
 4. The method of claim 1, whereinsynchronizing the audio stream with the video stream comprisescalculating a time duration corresponding to the set of audio data inthe video stream.
 5. The method of claim 1, wherein synchronizing theaudio stream with the video stream comprises calculating a total numberof audio data in the audio stream that matches the set of audio data inthe video stream.
 6. A computer readable medium storing a computerprogram which when executed by at least one processor processes mediadata, the computer program comprising sets of instructions for:receiving an audio stream comprising a series of audio data; receiving avideo stream comprising a series of video frames and a set of audiodata; and synchronizing the audio stream with the video stream based onthe set of audio data in the video stream by (i) determining acalculated sampling rate based on the set of audio data in the videostream and (ii) resampling the audio stream at the calculated samplingrate.
 7. The computer readable medium of claim 6, wherein the set ofaudio data in the video stream are duplicates of at least some of theaudio data in the audio stream.
 8. The computer readable medium of claim6, wherein the set of instructions for synchronizing the audio streamwith the video stream comprises a set of instructions for utilizing atleast one counter for counting the set of audio data in the videostream.
 9. The computer readable medium of claim 6, wherein the computerprogram further comprises a set of instructions for recording thesynchronized audio stream and video stream for synchronized playback.10. The computer readable medium of claim 6, wherein the computerprogram further comprises a set of instructions for multiplexing theaudio and video streams into a multiplexed media data.
 11. The computerreadable medium of claim 10, wherein the multiplexed media datacomprises: a self-describing section; and a plurality of interleavedaudio and video data.
 12. The computer readable medium of claim 11,wherein the set of instructions for synchronizing the audio stream withthe video stream comprises a set of instructions for utilizing theself-describing section of the multiplexed media data.
 13. The computerreadable medium of claim 11, wherein the set of instructions forsynchronizing the audio stream with the video stream comprises a set ofinstructions for indicating the calculated sampling rate in theself-describing section of the multiplexed media data.
 14. The computerreadable medium of claim 6, wherein the computer program furthercomprises a set of instructions for receiving a request to begin captureof the media data.
 15. The computer readable medium of claim 14, whereinthe request to begin capture is received through a user interface of anapplication program.
 16. The computer readable medium of claim 6,wherein the set of instructions for synchronizing the audio stream withthe video stream comprises a set of instructions for calculating a timeduration corresponding to the set of audio data in the video stream. 17.The computer readable medium of claim 6, wherein the set of instructionsfor synchronizing the audio stream with the video stream comprises a setof instructions for calculating a total number of audio data in theaudio stream that matches the set of audio data in the video stream. 18.A system comprising: a) a media source for providing an audio/videomedia stream; and b) a computer for (i) receiving the audio/video mediastream, (ii) generating an audio stream comprising a series of audiodata and a video stream comprising a series of video frames and a set ofaudio data, and (iii) synchronizing the audio stream with the videostream based on the set of audio data in the video stream by determininga calculated sampling rate based on the set of audio data in the videostream and resampling the audio stream at the calculated sampling rate.19. The system of claim 18, wherein the computer is further forreceiving a request to begin capture of media data.
 20. The system ofclaim 18, wherein the set of audio data in the video stream areduplicates of at least some of the audio data in the audio stream. 21.The system of claim 18, wherein the computer is further for multiplexingthe audio stream and the video stream into a multiplexed media stream.22. The system of claim 18, wherein synchronizing the audio stream withthe video stream comprises utilizing at least one counter for countingthe set of audio data in the video stream.
 23. The system of claim 18,wherein synchronizing the audio stream with the video stream comprisescalculating a time duration corresponding to the set of audio data inthe video stream.
 24. The system of claim 18, wherein synchronizing theaudio stream with the video stream comprises calculating a total numberof audio data in the audio stream that matches the set of audio data inthe video stream.